Orthogonal parallel MCMC methods for sampling and optimization

Monte Carlo (MC) methods are widely used in statistics, signal processing and machinelearning. A well-known class of MC methods are Markov Chain Monte Carlo (MCMC)algorithms. In order to foster better exploration of the state space, specially in high-dimensional applications, several schemes employing multiple parallel MCMC chains have beenrecently introduced. In this work, we describe a novel parallel interacting MCMC scheme,called orthogonal MCMC (O-MCMC), where a set of “vertical” parallel MCMC chains shareinformation using some ”horizontal” MCMC techniques working on the entire population ofcurrent states. More specifically, the vertical chains are led by random-walk proposals, whereasthe horizontal MCMC techniques employ independent proposals, thus allowing an efficientcombination of global exploration and local approximation. The interaction is contained inthese horizontal iterations. Within the analysis of different implementations of O-MCMC, novelschemes for reducing the overall computational cost of parallel multiple try Metropolis (MTM)chains are also presented. Furthermore, a modified version of O-MCMC for optimization isprovided by considering parallel simulated annealing (SA) algorithms. Finally, we also discussthe application of O-MCMC in a big bata framework. Numerical results show the advantagesof the proposed sampling scheme in terms of efficiency in the estimation, as well as robustnessin terms of independence with respect to initial values and the choice of the parameters.